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ABSTRACT 1 

Decision theoretic toting was used to explore 
whether junior high students could improve their tendencies to make 
realistic assessments of what ttt^y knew. In nine sessions over a 
period of three weeks, 49 sevenili graders used computer terminals to 
record probability values in the form ot logarithmic equivalents for 
each alternative in a randomized set of multiple-'Choice math 
probleas-. Students were assi^gi^ed to three treatment groups. One 
worked individually^ one worked individually with feedback from the 
cooputer^ and the third group worked as teams with feedback from the 
cotnputer. Analysis of the training sessions and posttests revealed 
differences in the stability of assessment behaviors but a common 
tendency to be overconf ident# to be less realistic as problems become 
more difficulty anfl to distort the value of one's knowledge when^ 
working as teams. Eealism t:raining was aost effective when explicit 
feedback was ^-rovided and the achievement level was low. 
(Auvhor/PC) 
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Decision theoretic testing is used to explore whether 
^ — ior high students can improve their tendencies to make 
realistic assessments of what they know, i.e. to assign 
subjective probabilities congruent with their information 
and reasoning about the problem at hand. In nine sessions 
over a period of three weeks, 49 seventh graders used computer 
terminals to record probal5ility values in the form of 
logarithirdc. equivalents for each alternative in a randomized 
set of multiple-choice math problems. \Ss viewed problems, 
calculated values on a set of panels callea a SCoRule,' entered 
a value for each alternative, and received immediate feedback 
after each problem. 

Students were assigned to three treatment groups. One 
worked individually, one worked individually with explicit 
feedback about observed realism in the form of a computer- 
-generated graph at the end of each session, and the third 
group worked as teams of size three or four, each team sharing 
a terminal and receiving the graph. 

A paper-and-pencil posttest showed a slight gain in 
realism over an equivalent pretest, but the difference was 
not statistically significant. A dramatic shift toward 
defined realism^ (observed realism within five degrees of 
perfect realism) did occur, however, among low achievers in 
the second treatment gr6up. A second posttest, administered 
as teams of size three instead of individually, contained 
harder items than the first posttest, and realism loss was 
significant, especially for the team study treatment group. 

Analysis of the training sessions and posttests revealed 
differences m the stability of assessment behaviors but a 
common tende.ncy to be o.verconf ident, to be less realistic 
as problems become more diffirnlt, and to distort the value of 
one's knowledge when working as teams. Realism training was 
most effective when explicit feedback (i.e. tlie graph) was 
provided and the achievement level was low. Time constraints, 
however, may have reduced effectiveness, and an extended 
program of realism training may be needed in order to modify 
long-established tendencies. 
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!• Staten.ent of the Problem 



Educators frequontly assert, that along with knowing 
facts, principles, and skills it is important that students 
know what they don't knov;,. A student who is unaware of the 
limitations of his knowledge is unprepared to make sound 
judgments in situations of uncertainty. By placing either 
too much or too little value on what he knows, he will make 
decisions that fail to take full advantage of his state of 
knowledge and may even load to serious losses. 

Above all^ the educational advantage of 
training people - possibly beginning in early 
childhood - to assay the strengths of their own 
opinions and to meet risk with judgment seems 
inestimable. The usual tests and the language 
habits of our culture tend to promote confusion 
between certainty and belief. They encourage 
^ both the vice of acting and speaking as though 
we were' certain when we are only fairly sure and 
that of acting 'and speaking as though the- opinions 
we do have were worthless when they are not very 
strong. (Savage, p. 800) 

One type of the "usual test" that may foster confusion ^ 
betv/een belief and certainty is the simple choice method in 
which a single alternative receives a probability of 1.00 
(certainty) even when the alternative is given only a slight 
pre'ference over the others. By excluding the other alterna- 
tives in che presence of uncertainty, the student learns to 
assign both more worth and less worth to what he knows than 
he should. He passes over the value of information that 
favors excluded alternatives and ho gives disproportionate 
value to information that supports his choice. 

Applications of decision theory to testing (Shuford, 
Albert Massengill, 1966) havd led to the development of 
'new methods of responding in which students distribute 
probabilities over the range of alternatives. Besides im- 
proving predictive ^^alidity and increasing the cpount of 
informatxon available to the teacher, decision- theoretic 
tests gc^neraliy require fev;er items than conventional 
tests. More important to this study, however, is the 
capability provided by d3cision-'theoretic tests for measuring 
student realism, a new dimen.^ion in student achievement. 
Realif^m is a congruence between the probability one assigns 
to an event and the probability that the event will occur 
given what one knows. 

A student witl) a tendency to make realistic assessments 
of his state of knowledge ana competence knows precisely 
the worth of his information and reasoning in making decisions 
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and solving prcblen.s. As. a result, hG recognizes his level 
of mastery of a topic. Ho s av.'are of the worth of hi^ 
judgment and aware of his Icick of critical information. He. 
knows when his rccisonLng logically tight c:nd when it is not. 

Measuring roalisui is analogous co assessing the accuracy 
of weathor forecasting. Probabilistic predictions of rain 
have become common in weatl^.er forecasts. If rain falls with 
approximately the frequency estimated by the forecaster, he 
uses his information realistically. Thus, predictions of an 
80% chance of ral)i (p^-.80) can be grouped into days when rain 
fell and days when no ram fell. If the proportion of days 
when rain fell to the total of the two groups approstimates 
.80, a data pojnt can be plotted on a 45 degree line indicat- 
ing perfect realism (Fig. 1). If, however, predictions of 
a 50% chonce of rain are associ^ated wir.h a set of days in 
which" the proportion of ra^n days was only .30, a point 
would appear below perfect realism. With enough data points, 
it is possible to plot probability assignments against . ^ 
relative frequency of rain and arrive at a loa^t-squares 
estimate of the forecaster* s realism line. Similarly, a 
student deviates from perfect realism to the degree that his 
reported probabilities fail to match his true probabilities 
as reflected by the relative frequency with which his knowledge 
allows him to ascertain the out-come of an event (e.g. the 
correct solution) . 

The relationship bctv/ecn realism and achievement is 
depicted by the inflormatiOh s quare (Fig. 2), in which the 
horizontal dimension is realism and the vertical dimension 
is achievement. VJhen the student's actual state of knowledge . 
lies beJow the level at which ho estimates or perceives that 
knowledge, the student is overconfident, in the sense that 
he overvalues what ho^ knows. V/li^n perceived knowledge is 
below actuai kno\%iedq>w he is underconf ident . To be realistic 
the student must exhibit a tendency to evaluate his knowledge 
at the level of actual knowled-.je. Note that perfect realism 
is possible at any level of c^chievement. As an interpretation 
of the information scjuare, Shuford ^( 1973 > cites this Arabian 
proverb: 





no 


> v/ho 


knowfc- 


cjnd knos^s th it he knows, 
is v/xse, follow him.* 


* 


He 


who 


khows 
' He 


?nd Rnow.? not that he knows, 
is aslcTsp, avs^aken him. 




lie 


who 


knowi; 
He 


not and knov;s not that he knows not, 
is a fool, shun him. 




He 


who 


knows 
he 


not and knows that he knov/s not, 
is a child, teach him. 



A<^ tho pio'^cib ^5ugcjos^S/ one consequence of realism may 
be more efficient learning sinco the student knows v/here to 
•concontrate hi-; efi'orts. A tendency toward realism may have 




Mas lory 
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Actual 
Information 




a nun\ber of other important benafits, both coqni.ti</'e and 
affoctivo-. . The^» realistic studonu has ioarnc^d to niak^ 
critical' ovolucitions ot knov' ] odqe cJaiiPS. tic has learned- 
to apprpach pr>ljica*i without, bias* Ho luv? rnade 'docisions 
that capj,taJj:rc on his av\-:ilabib resources, neither 'oxag-- 
derating noi: deprobatinv IhLir.. Under conditions o£ risk 
and uncertainty, ho manages ^9.^P what he knows in 
perspective* As preparation fpf'^'living in the uncerta.in 
context outside of scliool, realism tr'aining may have far- 
reaching significance. 
I 

Thc^ objective of th^is study was to examin(i» Lh^?- extefit 
to which realism training could bo effective with a group 
'of seventll grade math students. Realism training in this 
■instance was a scries of short but intensive sessions in 
/ma thoh.atical prolMen^- solving, ..using the computer to admin- 
ister and monitor decision-theoretic -testing and to provide 
imiuodiate feedback to the students. An effective trainma 
program is one that results: in a significant improvement la 
realism for many of t lie student^, y^c-^ording to the little 
data available, the degree of measured realism for junior 
high students appears to be relatively stabie-dver time. 
These fin\3ings, however, came from unsystemacic and noji-; 
statistical studies* It was the intent of this research 
to mvestiCjate a concentrated effort to improve students* 
assessment tendencies. y 

\ 

m 

II. Procedure 

The cxueriment was based on previous research with the 
"SCoRulo,'* a set of panels by which to translate probability 
assessmoots Into Ipgari thmic'^ equivalents (Shuford, 1970).. 
For example, on a four-alternative item v;ith a single correct 
answer, each alternative has a .25 probability of being correct 
if each 'alternative is equally likely. The logarithmic 
equivalent on the SCoRule, however, is 70. An alternative 
with only .10 probab.i Ij ty nov*^ ctheless has a SCoRulo value 
Oi; 50, whiir> the value for .SO probabiJity is 85 and .60 ^ 
probability is oquivaJent to b9. In otl^cr words, the 
logarithmic transformation returns propoXtiona tely large 
SCoRule values for lov;c r pi:of)abj^yli ty values but SCc^ule ^ 
values increase only slightly in the upper range of prouabilxty 
values. 

The student assigns SCoRule values to each alternative 
and receives the value ho has assigned to the alternative 
that is corx'ect. Because his score is a log value, the 
student loses Itttl^by slightly reducing the probability 
he attaches to the alternative he prefers, but .by slightly 
increasing lower probability values he stands to make a 
substanticJl SCoRnle gain if one of those alternatives turns • 



out to be correct. Tne SCoRulc scor iruj system therufore 
makes worthwhile to the .rtuciont to reveal his uncertainty. 



A* Subjoct.s' 

■ * 

T^.vo seventh yrado classi>s frvom a suburban junior hlcjh 
school participated durxna their regular 4!3-Runute math 
perLod>s. One class / tj.ught by the department chairman, v/ao 
•cons;idered higher in achievontonc tlian the otxier class. A total 
of 54 subjects participated in the experiment but because of 
absencqs and unexpected ca rcunus tnnce^- only 49 px'ovided 
adequate data for analysis/ 2b m the high-achie^'ing class 
and 23 in the lov^-achieving class, 

Mater .i als 

\ The experiment used six forms of the mothematical 

problen-solving sections, oT the Iowa Test of Basic Skills, 
Forms 1-4 were pooled and then divided into 15 levels of 
difficulty on the basis o£ item analysis information provided 
with the test. The fi.rst item at each level was assigned to 
the problem set for the first training session, the second 
item at each level to the problem set for the second session/ 
etc-, until nine probl(^m sets v;ore prepared for nine t3^aining 
sessions: Form 5' was admanistercd as the pretest and Fom\ S 
as the first posttest. A second posttest consisted of the 
last 21 Items in Form- 5, v/hich no S had reached during the 
pretest. 



At the time ':>f the pretef^t a SCoRule Procedures Test 
was adninistered Ln order to 'iioasure ability to use the SCoRulo 
after pretraining (Appendix A), 



^* Experimental Task 

^iine training sessions were conducted in the Computer- 
Assisted Instruction Laboratory at Stony Broo)^ during a 
three-week period, A\: each sesiiion S used a scope-and~ 
keyboard comput^-r terminal to register SCoRale values opposite 
each of four alternatives for as many items as time pormitted. 
The' computer monitored the input/ converting the SCoRule log 
values to probabilities. If the probaDiiities assigned to an 
item did not sum to 1.00 wxthin a tolerance of -04/ the computer 
re luired the st\ident to retype his SCoRule values. 

Because oi^ time required for busing to and from the lab, 
each session was 20-2 5 minutes Jn length. At the end of each ^ 
session the computer generated "realism graphs" Like Fig, 1 
with two exception-s. First, the vertical axis was changed so 
that an obr.erved realisn lane deviating from perfect roalisru 
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in the direction of overconf idonce csppeared with th*- upr^«^> 
part of the line cibovc Lhr pyrlo-l reJiLism iiiic, 

'-^underconf idcnce was graphed helow perfect realism. This? 
cTi5Trge made the graph easior to interpret to Ss since thoy 
could readily ^yee if they wore overvaluing or undervaluing 
their knowl^^Jgo. Second, additioncil feedback appeared on 
the graph: \ (i) one of the follov/mg words: "realJ s<;ic, *\ 
"overconfident , " "underconf ident; " (2) the number of 
problems completed; and (3) the accumulated SCoRuie score 
for the session. "Realistic" appeared if the deviation 
between observed realism and perfect realism (cf. Fig. 1) 
was five degrees or less in either direction. The decision 
to set five degrees as the tolerance range was the result of 
a series of simulations on a PDP~10 analyzing a variety of 
SCoRule settings. The simulation procedure, briefly, was 
to construct several 15x4 matrices, each representing 15 
four-alternative items and log (SCbRule) values that yielded 
an observed realism line witli a slope- of 1.00. Slight 

J changes in values were made until the settings indicated to 
the experimenter a clear case of either overconf idence or 
underconf idence. Departure from realism consistently 
^appeared when the angle between perfect and observed realism 
exceed©^ five degrees, or defined realism. 

Though em.phasis was on achieving realism, each session 
provided the cumulative number of items and the cumulative 
score at both the end of the session and after each item. 
f The purpose was to encourage S to work rapidly on a number 
of items rather than I'ngering over a single problem until 
extensive calculations removed all uncertainty. S was told 
to expect to be some^^'h^t uncertain in registering deci$?ions 
but to do as many items as possible while using what he did 
know as wisely as he could. The score v/as designed to dis- 
courage carelessness on the p^rt of those v/no v;orked rapidly 
as v/ell as to provide an incentive- for attempting to do all 
15 items. 

S^s were instructed thcv-t the preferred strategy in 
attacking a problt^ra .vas to s^^t the SCoRule to indicate no 
knowledge, i.e. each alternative with .25 probability/ or 
SCoRule values cf 70-70-70-70. ^-'alues could then be raised 
or lowered depv^nding pn S^'s luiowlcdgo with regard to given 
alternatives. The cumulative score -vas also designed to , 
discourage S fron. simply lea^'ing the SCoRule at the no-knov/ledge 
level, thereby showing excessive caution. 

/ 

iVhen ay graph appeared at the conclusion of a session*, 
the experimenter attempted to interpret the results v/ith 
comments such a^i, "'Overconfident' means that you are placing 
too many points on v/hat you consider the best alternative," 
" ' Undeicontidcnt ' means that you need r.o give more points to 
the alternative you think is best," "'Realistic* means that 
you are giving juf t about tJho right cimount of value or your 
SCoRule to each alternative." Suggestions were sometimes 
made to "sprriad out points" (overconf idence) or to "iiunp 
points" (underconf idence) • 
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with vin if ip^ o/r nt :*\ :i I $ ;-n^ Ino udy usx.d u pretext 

oloccLi •( ^/a.-- ^ <j AoL:.i\ijric ./p^' tnc r <u\y chan-jc v/ould occur, 
Leav i iis^, to a i a te ^ f t l v coi :\ r i .^n:? .i\h con l:rt:> I '^roup . 

ixic I* re o -'t aru'l r':.'3t p •;::rLt:o£l: vore both taken 
iivi ivi dual L- t]^ th., nJlcr" descrj oed above. The scvoi'i<3 
posii t. I v;-; b aan i a : »^ o l\ d Si. wor ^: inq a ^ Con^tS of tlirae 
on rh^v idv e-llov.'L;ia U.c f r po.-ttent. Thou^''*! tboy wore 
en-.^ourage:! to cori^^ult v. 5 th oti^cr ro-ini luembcrs, ^hey 'vere toid 
t.'> »ocoii s::^^:<'ii » v.Mu.^t. i,ep irately, nu^kincj choir ov;n 
accisions. noved carlior^ Uio piobieins on the second 

posttost '.vero roro difficu\t J-har; thos^- on the otiier tv.'O 
ttsti.. T -.iiLo vcrc io^ruc^; by tn^ ..eachers ixi order to 
c^ocur^' CIS uvi. par*-.i cipcition "ind cooperation in the groups 
as" post J bie - 

7i secop.dary objective , bo.-.evor, >as to examine the 
relative effectc of. exposure to tho graph and team problent- 
solv'ng. Throo qronp.s //e: <i forrr. ^d m each class. Hjie groups 
diitorcd : n t lie t\ po cf support axvcn during realism tr;?.ining. 
Group 1 recej'Gd Lhr least suipport. Group 2 received explicit 
feedback (th-- ^re^ph) and Gro^.p 3 (team study) both explicit 
feedb?.ck -^in^ peoi* ^ncccactjon. v:hethcr or not peer interaction 
v/as m f-irt srtp>ort)ve in atla n ,ng realism v;as one quei^tion 
the stud" -;wUGbt :^ oy i a. . 

Each teaciicc ns^.^iiUiC on»_ third of his class to the tean* 
study qrouo in ordox t j k-^en ^ach team congenial. Insofar 
as po<^SLbi( , tccichcrs rn'-^ each rcproscmtative of the 

class, '-riih h\ f i lor . overs be Ltrceu by low achievers. 0/iC 
three-f^Ci'ijLer teaT^ .n f-ach claf^s v/::s male and one fcf.iale. An 
oddi'. icnal fcrale ^-:-c.nt in the- Iciryer class had four ^'Onibcrs, 
one of '^mo'^ .Tiroli.t .i 'i*"t<-r tho voriginal asnignmenis vett^ mad'^- 
but before t!ain?n<: hcj m. 

T\£» ri^^tainin j tv:' ir.irdG v the total sarpie v;.:re di»/a.dod 
into four rLrv_*:.^ i^ioro n.^>nc? .-ondoinly assiqned to v;orK aiopit- 
^:thor witii or m ».0'..' .^xp;;J.'o t'. tho urapii. The pcet<=»st. vas 
useo to Fia :e a f^ol^'n f.irv ^.-i:- iii-.*it^«r of achj c^'Oinenl level oy u.s..> 
t^.• ni?:(')r^L o." : t«.»r's c'^-i;pictc^ "^^ un index, siiice prv-tpst ricorco 
oonld not t 'If n l-r Oi^'.an o^i ".'^thou^ I ^ny and c^; rtolica ted analy.^'' s 
(Sin.-x- then ^ I" :vi'P'-'J pi:o::ru/:i :aUed bEC-TEST (Brennan, i97i) 
has be^;n v/r-^t'Cn by - I: i \ '>n.^nn.jn Slony Brook by which 
to ma':o rapid .in.ilyst^ d^^vi Mon-thoore^t ic tests.) 'JTk» 
index cor I'o'l .0^ is ..r-^ Tivth vipp.! ical i ons section of 

tho Caliiorti,<» 7*.Jh e'/ P' nt ]'o£:t., -./hic)! .-^as available tor only 
.>v" >r fhr - , ;i I'f , ^he r.^jlinq i^roVL- ' h^tt 

t'.s-^, -..rij^Mi A-'i . d-'-i.-LL*:- lot . (The Ln^o part^. ci tiio tc>-t 
C( rrc 1. te^: . . 



^ ' i.^ic*^.u_^ Wjtcii ci}e bcoi^ulo nucjht also influence 

the effectiveness of realism training another index was 
computed us^n<, i'C'>i- *i ' )^ ' I ^ 

Test'* developed by unc 5xpcr j.iUt-iiU<,M: (Ai^penUix A; • Cox xciaLxi.ii 
bolv/con this* fam: liarity index ^and the number done, the math 
:applio:it, .n.. y^' \ un i iP,':r c \^cro .10, .20, and ,24, 

The nican nuiober of ^ tcn^s conpjloted on( che pro t-o sc. was 
2U.G, Tne an fCoilui.: u si score 7.:j. if ^ wui. aoove 

the :r4Gan in both jases, he- or she v/js as.signed to'the hicjh- 
achir-wr,/:!' .jh-SCo'-^.»lc iD.iiiariW oIocm, Those belo^/ the 
mean wr.v^ con.-.i-Jc rcc lov;, t i.t],er in achievement level or 
SCo.^uio fami^ ctn*, or Doch, Wi.thin the four blocks chat 
v.erc: I'orrio.d, ./r-co cil.^o b!ock*:d by class and by sex. 
Those m /Ihc ip^fli cl<^ss (c^^jls m and C in Fi<5ure 3) were 
randoaay assicjrcd t.o .vne of the two individual study treat- 
rrcritc, with thv> numhe}' o"" males in each cell kept ecrual, 
Si.T.ilarl^- Ik.^ ciri^B Ss '.-/e^^'e randomly ass.igned to ceils B and 
n. 1-igure 3 sho-..s cell Ns, sex com.pos i ti on, and mean scores 
on blocking ::vi cos for = he full 2x3 design. Analysis of 
variancf. disclo.-.ed that the treaUiients did not differ on the 
ach:^:t:vcrhent 'li.do^x (F-'l,26j, nox did die cells ditfer on 
either CAT i -^tii appti ci*^. ion^^ scores (F=.56) or reading scores 
{F=,19). ""^jic-f iy si of the SCoRule t'aauliarity index revealed 
rore variability (r=3.34, p<.0-3), although individual study 
trcc*tr.:onts ^ere s.mi Ia» {F^ . 31) . 



III. '<OoUl ti^ 



Pre te 3 1- Pos t lo l Con r x. s n ^ ■ 

The n;aio: dependent var. was the degree of the angle 

becv;oen obscr-rd -oniis^. ^.^rrioct realasm. Observed 

realism -ya^ roi-iLu^ci ] y m^. procoJurc- described in Appendix 
B, Table I o-c ^-.nt^ tic rtean ancj Ic of deviation f or^ e^ch 
trcatn:en- <^rjup .s^.i t )s' the tatai saraplo on the pretest and 
the two po.:Lt': 5 . Tl r r'.?ult<~^ ^.s.sf^n tialiy no dii ference 

between p>-i>t^'5?, and pc/t-i.-st ^)oh;>'. io>. except for Group 3 oa 
the secor^.5 r,,.s'. est. 'Jr.ouq^ th- o'^erall mean cind the means 
for Groups 1 .Mid : ::hc-^ -i 3i:ght gam in realism on Pos..ttcst 1, 
the Wilcoc^-^n s i gn^- i--ran^'s \or\. foj- tv/o matched samples failed 
to d^-ciabo sign ijcai^ci^ (z^.OO, ,1), and 1.17 respectively). 

Vvhile the Kjn i ' ud' • o"" realism gain on the first post-- 
test IS smctil, Vh'.- dj rocr ion of the change/ in standard 
devia t i on^^ i :j cons 1 ( en tly tov/a rd improv^^r/en t , Even the 
second po^-ttes? , v.hich combined student tx^ams with harder 
problems and showed a shajp loss m reali^^m, indicated tlvat 
variability decj eased fcJ/Jowing training.^' 

An estiriato of the proportion rorxoLt can be obtained 
by consxri ";rj !Kj the mo.st hijhiy /.jLuod alternative as the 
single response G v/ould have made undo>' classical testing 
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B 



0.:. 

18 78 "P 



N=8 
M-3 



6.3 



N=--9 
M=3 
F--6 

F-5 



7.8 ; 
19.0* F 



B. 1' 



TEAM 
STUDY 
GRAPH 
3 



23.1 



Nf=10 

M=^3 

F=7 

N=6 

M=:3 

F=3 



6. 
'23. 



0 



3.4! 



Figure 3. S iPiplc nn-e, ne^in number of 
itt-nis complc tc'l on pretest 
(upper rit.'h\ rornor) , and 
mcMii 3Cor' - '.in SCo Pu Lg Proced- 
urf;.T Tt f;t (i v.vr-r r.icjht corner) 
Hi oacii cf'i ••>': t3io dosiqn 
used to.. .r-^ Ireatmont 
oir(.'-tc-' (p.-rtale, F--t'cina] o) . 
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lable 1 



Troatirient 



Mean Angle of Doviation frou\> Perfect 
Realism on Pretest and Posttests 
By Treatment Groups m Degrees 
(Standard Deviation in Parentheses) 



Group i 
(individual study 
witliout graph) 
N=i5 



f 



14.9 
(II. 0) 



First 



Realism 



Second 



> 

Realism 
Gain/Loss 



Pre tc-st Posttest Gain/loss P os ttest over Pretest 



11.1 
( 7.2) 



43.8 
(+3.8)' 



17.4 
( 6.2) 



—2.5 



Gr9UD 2 14.4 10.8 +3.6 18,1 -^3.7 

(individual study ( 9.9) ( 9.1) (+ .8) ( 7.7). (+2.2) 

with graph) 
N--18 



Group 3 
(team study ^ 
with graph) 
N=16 



10.6 
(12.6) 



12.2 
( 9.^} 



-1.6 
(+3.2) 



22.2 
( 9.6) 



-11.6 
(+3,0) 



Total Sample 
N=49 



13.3 
(11.3) 



11.3 
( ^.7) 



(+2.6) 



19.3 
( 8.3) 



-6.0 
(+3.0) 



p ^.01, V/ilcoxen aigncd-ran):*? tcnl 
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conditions, i.e* tho choice method. In the case of Lied 
v^nlues/ one is picked at random xn orde>' to estiiricatc S^'s 
response. Table 2 rcoals that the pretest and first 
por^ttest wore of comparnbio dLffLculty but the second 
postto^'b was consi dorctl^ly harder. Varaabilxty decreased 
on the ppbLL'v^sts as it did with respect to the realism 
anqle. ' 

,Kstiinates of proportion correct confirmed tho teachers' 
judympnt tliat the tv/o clasboi=; .di f f ered m achieveir«ent level* 
The l(J?w cla.s5; renui Lned nt least lb points behind the high 
class but showed a modest relative gain (Table 3). 

What gain in roalasm that ^id occur on the. first 
postlest came primarily from the low-achieving class, as 
shown .in Table 4, Although the gain of 4 •2\ degrees was not 
statistically significant (z--=.70), the overall pattern 
indicates that the low class cended to respon\ more favorably 
to realism training than did the high class, l^he pattern also 
appears when the classes are compared according\^to the 
proportion in each class whose observed realism line fell 
within five degrees of tho perfect realisin line , \<e, the 
proportion who were "realistic" witliin the range of tolerance 
used in the experiment ( defined realism) . In the high class 
the proportion was the same for both the pretest and the first 
posttest^ while the proportion m the low clasfe more than 
doubled (Table 5). 

The proportion v/ho attained defined realism increased 
the most for the group v^rorking individually and receiving 
the realism graph (Table 6). Team' study v;ith the graph, 
however* was associated v/ith a decrease. Substantial posttest 
gain r'eally occurred in only one cell in the original design: 
low achievers itnder individual study with the graph (treat- 
ment 2). Table 7 presents the distribution of ^s on the 
pretest and first posttest in the three categories that were 
used for feedbdok with the r^inlisnj graph'. tvliile the number 
achieving defined realism is stable from pretest to posttest 
for fiv^^- of tho six cells, und..»rconf jdent and overconfident 
students move shdi.ply tov/ar i re il i ski m the low-class-graph 
feedback group. ^ ' 



Traini ng Se^-s ion Trend/' \ 

Keali ^i^in ajxl diiticu ^Lv . ^"stimates of realism during 
the C-A.I. sessions arc based on fewer units of analysis 
because of * t'vo major factors: absenteeism and the fact that 
Group 3 sh:ired terminals. In general, S_s shov;ed less 
realistic bc^hovior during the njine sessions than on tho pre- 
test ana the first posttest, but mean fluctuations remained 
within the range of realism taounded\by the means of tho two 
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Table 2 



Mean Estimated Proportion Correct 
On Pretest and Posttasts 
Dy Treatment Groups 
(Standard Deviations in Parentheses) 



Treatment 



Pretest 



First 
Posttest 



Second 
Posttest 



Group 1 .62 
(Individual study (.18) 
without graph) 
(N=15) 



• 61 
(•13) 



• 48 
(.11) 



Group 2 .55 
(Individual study (.21) 
with graph) 
(Nxl8) 



.60 
(.17) 



.52 
(.11) 



Group 3 
(Tocun study 
with graph) 
■(N=16) 



• 66 



.65 
(.16) 



.46 
(.12), 



Total Sample 
(N=49) 



.61 
(.20) 



.62 
(.16) 



.49 
(•11) 



Table 3 



Mean Es Li mo ted Proportion Correct 
on Protest and Posttesta 
By Classes 
(Standard Doviatioii in Parentheses) 



Class 



Protest 



First 
Posttest 



Second 
Posttest 



High 
(N=26) 



.70 
(.15) 



.69 
(.13) 



.56 
(.08) 



Low 
(N=23) 



.50 
(.20) 



.54 
(.15^) 



.39 
(.00) 



Total Snmple 
(N=49) 



.61 
(.20) 



.62 
(.16) 



.49 

(.11) 
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TnbJo 4 



Moan AngJe of Deviation from Perfect 
Realism on Pretest and Posttosts 
By Classes in Degrees 
(Standard Deviation in Parentheses) 



Realism 

First Realism Second Gain/Loss 

Class Pretest Posttest Gain/Loss Posttost over Pretest 

** 

i 

High 9.2 9.1 + .1 16.2 ' .. * 

{N=26) (7.5) (5^9) . (+1-6) ( 4.4) (+3.1) 



Low 18.0 13.8 44.2 23.5 --5.5 

(N=23) (13.0) (10.5) (+2.5) (10.3) (+2.7) 



Total Sample 13.3 11.3 +2.0 19.3 -6.0** 

{N=49) (11.3) ( 8.7) (+2.6) ( 8.3) . (+3.0) 



p < .01, Wilcoxen signed-ranks test 



Table. 5 

Proportion Attaining Defined Realism 
(Within 5 Degrees of Perfect Realism) 
By Classes 



Second 

Class ' Protest Posttest . Posttest 

^Jig^ .35 .35 .00 



(N=26) 



?ow .13 .30 . .00 

{N=23) 



Total Sample .25 .33 ' * • .00 

(N^:49) 
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Proportion Attaining Defined Realism 
(Within 5 Degrees of Perfect: Realism) 
By Treatment Groups 



Treatment 



Pre test 



First 
Posttest 



Second 
Posttest 



Group 1 

(individual study 
without graph) " 
(N=15) 



.13 



.20 



.00 



Group 2 

(individual study 
with graph) 
{N=18 



11 



39 



00 



Group 3 
(team study 
with graph) 
(N=16) 



50 



38 



00 



Total Sample 
(N=49) 



.33 



00 
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Table 7, 



Distribution in Three Categories of Observed Realism 
By "^reatnient Groups within Classes 
on Pretest and First PosJttest 









Pretest 






First Posttcst 




Treatment 




Under 




« 

Over- 


Under 


Realistic* 


Over 


Group 


N 


conf . 


Realistic* 


conf . 


^ conf. 


conf 


1 


7 


1 


1 


5 


0 


2 


5 


High 2 


9 


0 


.2 


• 7 


o' 


2 


7 


Class 
















3 


10 


0 


6 


4 


0 


5 


5 


All 


26 


1 


9 


16 


0 


9 


17 


1 


8 


1 


1 


6 


v2 


1 


' i 


2 


9 


2 


0 


7 


1 


5 


3 


Low 
















Glass3 


6 


0 


2 


• 4 


0 


1 


5 


All 


23 


3 


3 


17 


' 3 


7 


13 


Total Sample 


49 


4 • 


12' 


. 33 


3 


16 


30 



♦observed realism within iive degrees of perfect realism 
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posttests (Figure 4). Both tITe smallcV sample sizes and tlie 
small number of iteras per session may i^e responsible for much 
of the fluctuations throuqh mcasuromenb error (sec Appendix 
C for session data and for the number of itoms completed 
during both tests a»u^ sessions*) . 

Fluctuations, however, do follow a consistent pattern 
if they are viewed m connection with'^he* difficulty of the 
items. .The estimated proportion correct across all ^s also 
fluctuates within the range bounded by the lueons of the tv/o 
posttests (Figurfi 0). Realism and difficulty, however, 
fluctuate in opposite directions, as shown by Figui?e 6. 
Increased diff.iculty is associated with less realism. 

Most of the fluctuation occurred in the low class, which 
was consistently less realistic than the high class (Figure 7). 
Though the high class showed a slight tendency toward greater 
realism, its performance was generally stable. Both classes 
exhibited variation from session to session in the estimated 
proportion correct (Figure 8). The low class consistently 
scored below the high group and peirformed less stably. 
Neither class scored especially high; even the high class 
found the tests. and sessions difficult. The strong re- 
lationship between realism and difficulty is apparent in 
Figure 9, which shows that for tlie low class item difficulty 
and realism angle were nearly mirror images. When the 
randomly selected set of training items was relatively 
difficult, ^s were relatively unrealistic, but realism over 
a session improved when items were less difficult. 

The comparison of the performance of the three treatment 
groups during training sessions is complicated by the small 
N'in Group 3 that resulted from sharing terminals. Group 1 
(individual study without the c/raph) exhibited the most 
realistic behavior until midway through thew^xperiment , when 
Group 3 (team study v/ith the graph) took ther lead (Figure 10). 
These two* groups , however, also tended to find the items 
less difficult (Figure 11). The realism and difficulty 
curves of Group 2 are another case of a near mirror image 
(Figure 12 ) . 

The effect of difficuicy on realism was also examined 
by comparing easy and h^rd it.ems that appeared during training. 
Because of random presentation v/ith in the problem set for 
each session, it was possible to divide each set into comparable 
subsets: items from the easiest seven levels and items from 
the hardest seven levels. While in a few instances did not 
encounter items from one of the subsets, the number of Ss and 
the mean number of items actually presented were approximately 
equal for the two subsets on a given training session (see 
Appendix D) The mean realism angles, however, were highly 
divergent for the two suDsets (Figure 12). Except for the 
third session, deviation from realism wa-s consistently 
greater for harder i terns than for easier items . 
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Figure -v. Angle between ideal and observed realism 
for total sample (see Appendix C for Us) . 




^•^^ SESSIONS Posttor'ir 

Figure 5. Estimated proportion correct for total sample , 
(sec Appendix C for numJ^er of items). 
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Figure 6. Relationship between realism (broken line) 
and difficulty (solid Jine). See Fic;ufes 
4 and 5 • 
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Pre- 
test 



"1 S 6 

Si;.\iSIOMS 



PosttcsLs 



Figure 7 . Anc,lc br^twccn ideal and observed realism 
for high cla^s (solid Ixnc) and low clasj: 
(broken line). 
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Posttests 

SESSIONS 



Figure 8, Estimated proportion correct for high class 
(solid line) and low class (broken line). 




Figure 9. Relationship between renliiJin (broken lino) 
and difficulty (^solid line) for low class. 
See Figures 7 and 0» 
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Pretest 1 



5 6 
SESSIONS 



Ti i T2 
Posttcsts 



Figure 11. 



HGtininted proportion correct for treatment groups 
(l=closed circle. 2=open circle, 3=trinngle) . 
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Figure 12* Relationship between realism (broken line) 
and difficulty (solid line) for treatment 
group receiving individual study with the 
realism graph (see Figures io and 11)* 
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SESSIONS 

Figure 13. Anglo botwocn icloal and obso vcd realism 
for easy items (solid line) and hard 
items (brohon line) during training 
(see Appendix D for Ns and ir.ean number of items) 
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Realism gains . Despite wide variations in performance 
•from session to session. Figures 4, 7, 'and 10 all suggest a - 
developing tendency toward improved realism. The overall 
standard deviation on tlie first post test was smaller than on 
the pretest, with the mean emgle of Session 9 at the oame level 
as the posttest (Figure 4). The two ciacjses were closer in 
realism performance by the time oL" the first posttest (Figure 
7), and treatment, group? tended to converge over the training 
sessions (Figure 10). 

Although the larger variability and generally larger 
realism angles in zhe early sessions may suggest that ^s 
were less familiar with the SCoRule than later in the exper- 
iment, the proportion of Ss having realism angles within five 
degrees of perfect realism was actually largest in the first 
session (Figure 14). Moreover, the proportion was relatively 
constant except for sessions 6 and 8. When the proportion is 
examined by classes, the curve for the low class tends to 
rise, but the curve for the high class tends to fall (Figure 
15). While the two classes finished with a similar proportion 
attaining defined realism, it was because the low class gained. 

The three treatment groups differed widely in ^the propor- 
tion attaining defined realism (Figure 16). The twQ individual 
study groups tended to move in opposite directions, the one 
without the graph having a higher proportion in the early 
sessions and the one with t]ie graph performing more realis- 
tically in later sessions and on the first posttest. The 
curves of these two grpups bear similarities with those of 
the high and low classes, especially with respect to the 
crossover that occurred midway through the experiment. 
Figure 17 breaks apart the curves for''' the individual study 
treatment groups in order to compare^ high and low class 
proportions in these treatments. Except for the sharp jump 
on the first posttest {no ^ was "realistic" on the second 
posttest) , the low group with the graph exhibited little 
variabilicy but upward spurts began to appear after the fir^t 
few training sessions. Th4 spurts resembled the more pronounced 
variation in the high group Hvxtb the graph, which also appeared 
late in training. 

The four groups of Figure 17 appear in Figure 18,' to show 
that the pattern toward realism is not so apparent when the 
mean angle of deviation from perfect realism is considered. 
Some relative gain in realism for graph groups over no-graph 
groups is suggested by the downward slope of the high graph 
group and the catch-.up pattern of the low graph group, but 
angle of deviation was not as indicative of improvement as 
proportion attaining defined realism. 

Over confidence 

The direction of deviation from realism was almost 
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Figure 14. Proper ticnVof total sample whose observed 
realism was within five degrees of perfect 
realism. 




SESSIONS Posttests 



Figure 15. Proportion of high class (nolid line) and 
low class (broken line) whose observed 
realish wa.^ within five degrees of perfect 
reaJ ism. 

22. 



3^ 





SESSIONS Pouttests 



Figure 16. Proportion of ti-catmont group 1 (colicl 

circles), 2 (ojion circles), and 3 (trinngles) 
whose observed realism was within five 
degrees of perfect realism. 
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Pretest i 2 3 4 5 6 7 8 9 First 

SESSIONS Posttest 

Figure 17. Proportion attaining defined realism in the 

individual study treatments: low-class-graph 
(open" circle) , lov/-class-no graph (closed ^ 
circle), high-class graph (open triangle), and 
high-class-no graph (closed triangTo). 
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uniformly toward over confidence throughout the experiment 
(Table 3). Of t*ho.?e ^^wi th unrealistic behavior, only four 
tended toward underconf Idenco on the pretest, only three 
on the first posttcst and only two on the second posttest. 
The proportions during training sessions were approximately 
the same as on the teste. The seunple showed a strong tendency 
to overvalue'^ information, i.e. to assign too much weight to 
what was mistakenly believed to be the correct alternative. 
Deviation from realism was especially strong on the second 
posttest, nearly all of it toward overconf idence. 

The bulk of those who were underconf ident during the 
experiment were in the low class CTable 9). Of the three 
treatments the individual study group with the graph tended 
^o do the most undervaluing of information (Table 10). The 
least undervaluing occurred in the team study group, in 
which all lack of realism was in the direction of overconf idence. 
Team testing (posttest 2) was also n^arked by little undervaluing 
despite the strong deviation from realism. 

In general, few ^s were consistently able to refrain from 
temptations to mass their probabilities on the wrong response. 
The overall pattern indicates that _Ss commonly placed more - 
value on their knowledge and their reasoning than their 
performance juiatified* Since lack of realism was nearly 
always in the direction of overconf idence, both realism and 
overconf idence appear to be sensitive to problem difficulty. 
Overconf idence increased along with difficulty in nearly all 
cases. 

One excep^^ion is tlie tendency of the low class to have a 
higher pro'porti o.*) of und erconf ident ^s than the high class, 
even though the high class found the problems less difficult* 
A similar exception is the contrast betv/een treatment groups: 
the group which had the most difficulty with the items 
(individual study wxth graph) had the highest proportion of 
under conf ident ^s. These except "ions suggest tliat underconf id- 
ence and overconf idence vvcre both sensitive to problem difficulty 
but the higher the level of achievement the more likely S 
deviated from realism in th^ direction of overconf idence. 

Another factor contributing to overconf idence appeared to 
be team discussion. Without exception^ any team that deviated 
from realism during training sessions exhibited overconf idence. 
No team member in Group 3 showed underconf idence on any of the 
tests, and only two Ss from the entire sample undervalued 
information on the teamed posttest. 

IV* Discussion 

Although ^^vidence from this experiment fails to confirm 
the general effectiveness of realism training in modifying 
assessment tendencies, tha study produced several findings 
about assessment tendencies of the seventh graders who 
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Table 8^ * / 

Number of in Three Categories of Observed Realism 

(Total Sample) 



Underconf ident Realistic Overconfident 



fie ues u 




A 
** 






Session 

1 


• 


2 


11 


17 


2 




4 


7 


15 

* • 


3 




2 


10 


22 


4 




2 


9 


20 


5 




0 


10 


24 


6 




2 


6 


25 


7 




0 


7 


20 


8 




1 


2 


22 


9 




3 


10 


18 


Posttest 


1 


3 


IC 


30 


Posttest 


2 


2 


0 


40 


*reaiism 


angle 


withirx 5 degrees 


of perfect 


realism 



/ 
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Table 9. , . * 
Number of S^s in Three Categories of Observed Realism 
' - By Class 



Underconf ident "Realistic"' Overconfident- 

High Low High Low High Low 

Pretest 1 \ 3 9 3 16 17 




Session 

1 1\. 1 7 4 8 9 

2 1^ 3 4 / 3 9 7 

3 • 11 5 5 ■•- 11 11 

4 0 2 5 4 12 8 

5 0 0 6 4 12 " 12 

6 0 2 3 3 12 13 

7 00 ^ 25 812 
a 0 10 2 14 8 
9 0 3 ^55 10 8 

Posttest 10 3 9 7 17 13 

Posttest 2 0 2,00 24 16 

♦observed realism within 5 degrees of perfect realism 
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Table 10. 

Number of Ss in Three Categories of Observed Realism 

By Treatment Groups 

Underconfident "Realistic"* Overconfident 

I 

Pretest 

iSi 

I 

2' 
3 
4 

5 : 

6 
7 
8 
9 

Posttest 1 
Posttest 2 

♦observed realism within 5 degrees of perfect realism 
**l=individual study, no graph; 2=individual study, 
graph; 3=teajtn study, graph' 
***The unit of analysis for Group 3 during sessions is the team. 
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participated in the experiment:* In addition, the data in- 
dicated that realism training was more, effective for low 
achieving students, and a pattern emerged among low achievers 
favoring individual study with ^ realism ^raph as feedback 
over no graph or team study* In view of the circumstances of 
the experiment, these findings suggest that a more extensive 
training program is needed before drawing the conclusion that 
realism cannot be substantially modified. 

Earlier v^ork of "an unsystematic nature has indicated 
that assessment tendencies of junior high students appear to 
be stable over time and may not respond to training. In this 
study, the proportion of students whose observed realism v/as 
within five degrees of perfect realism increased from pretest 
to the first posttest by only 8%. However, every sub-grouping 
of the sample showed a decrease in the standard deviatioi^ of 
the major dependent variable (angle of deviation from perfect 
realism). Certain groups, such as the low-achieving class and 
the individual-study treatments, responded more favorably to 
training than other groups. In some cases, notably individual 
study with the realism graph as feedback, a distinct trend 
toward realism occurred over the training sessions. 

Student Characteristics 

The effect of training was net as evident from this study 
as several characteristics of students' assessment tendencies. 
First, the tendencies of some students were much more stable 
than those of others. The low class in particular fluctuated 
widely in marked contrast to the steady behavior of the high 
class. Second and most prominent in the findings, students 
were characteristically overconfident in their use of infor- 
mation. The slope of observed realism nearly always fell 
below perfect realism when the angle of deviation exceeded 
five degrees. The tendency to place more value on an alterna- 
tive than one's information about that alternative is actually 
v/orth appears to be relatively constant in the data. Despite 
increased opportunities to use the SCoRule and gain more 
familiarity with the process of assessing subjective proba- 
bilities, students overvalued information toward the end of 
the experiment in about the same way they did at the begin- 
ning whenever they were not being realistic. 

Third, students became less realistic as problems became 
more difficult. The wide fluctuations in observed realism 
that occurred in some of the data are regularly related to 
changes in difficulty level. The relationship, however, does 
not appear to be linear because the size of the fluctuations 
was much greater when problems were especially difficult 
(e.g. the low class) than v/hen difficulty was reduced (e.g. 
the high class) . 
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Fourth, the data suggests that difficulty level may be 
related to overconfidence* In groups »for which problems were 
relatively less difficult, nearly all students v;ho failed to 
achieve defined realism exhibited overconf idence, while most 
of those few who were underconf ident v;ere in groups that had 
relatively more difficulty with the problems. 

Finally, overconf idence may also be related to team 
discussion. The absence of underconf idence in the team study 
group and the nearly unanimous expression of overconf idence 
on the team posttest may indicate that the dynamics of group 
interaction promote a tendency to ignore the limitations of 
one's information. The group, for example, may function to 
divert an individual from recognizing what he knows by subtly 
undermining his respect for his own judgment or by introducing 
extraneous personality factors into the decision-making pro- 
cess. The practice of assigning students to congenial team 
groupings may have contributed to a tendency to adopt the 
position of peers despite misgivings. 

It should be noted that the overconf idence so character- 
istic of this sample is not to be confused with an exaggerated 
sense of self--worth or an emotional state. Overconf idence, as 
Figure 2 indicates, is a cognitive construct dealing with use 
of information. It may be that affective characteristics play 
a major role in how information is assessed, but the primary 
meaning of overconf idence should be interpreted in the context 
of decision theory (de Finetti, 1937; Raiffa, 1968; Ramsey, 
1931; Savage, 1954). It is when, a student attaches value to 
his knov/ledge and reasoning about a problem in excess of the 
value his knowledge and reasoning are really worth that he 
becomes overconfident. It :s altogether possible that a 
student has a weak self-concept or appears unsure of himself 
while his observed realism indicates overconf idence, since the 
distortion reflected in his deviation from realism is oriented 
toward information rather than toward interpersonal relations. 

At the same time it must be noted that a relationship 
between overconf idence and personality characteristics is a 
least suggested by the data. Risk-taking tendencies and peer 
interaction are both important ingredients in decision theory, 
especially in relation to group problem-solving. If difficulty 
increases to a point at which the individual feels personally 
threatened, information distortion is likely to be in part a 
function of self-esteem. Whether he will express overconf idence 
is problematic, hov/ever, because individuals differ. Some 
react more cautiously eis difficulty mounts,. e»g. the under- 
confident students in the low class. Others become overconfident, 
refusing to acknowledge that they know less than they did on the 
easier problems. Self-concept may interact with assessment 
tendencies even more when interpersonal relations are part of 
the decision process. Members with weak self-concepts may 
acquiesce while their partners interpret acquiescence as a 
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signal to weight suggestions more strongly than tliey should. 
Information use is distinct from affective characteristics 
but probably closely related 

Realism Training 

Besides the general reduction in the standard deviation 
bf information distortion (angle of deviation from perfect 
realism), the data supports a tentative conclusion that 
realism training has beneficial effects in some instances 
despite the lack of dramatic gains overall. The low class / 
made steady gains in the proportion of students who were able 
to achieve observed realism within five degrees of perfect 
rbaliV^n. In later training sessions, this class even sur- 
passed the high class, and on the first posttest showed a 
gain while the high class proportion remained at the pretest 
level (Figure 15 )• " | 

The source ^of this gain within the low classj came ex- 
clusively from the second treatment group, in which irrlividual 
study was combined with the realism graph. The sliarp increase 
is even more striking in view of the lack of gain in the other 
five cells of Table 7. Although neither the prop|ortion attain- 
ing defined realism nor the angle of deviation frpm perfect 
realism indicated dramatic progress during the training sessions, 
the graph group receiving individual study in the low class did 
appear to make spurts in the direction of improvement 
(T?igures 17 and 18) . 

Realism gain may require a combination of feedback about 
realism and high problem difficulty. The apparently strong 
relationship between difficulty and realism may mean that . 
students are more likely to ^discover their long-es tab li shed 
tendencies to distort the value of information when they 
confront a high level of uncertainty than when they have to 
deal with uncertainty that is only moderate. The discovery, 
moreover, may require explicit feedback about their tendencies 
in order to establish an intuitive awareness of the subject - 
ive responses that contribute to realistic behavior. 

If explicit feedback is important to attaining realism, 
it may seem strange thac the angle of deviation from perfect 
realism fails to distingui sharply between the two individual 
study groups in the low class. It should be remembered, however, 
that explicit feedback v;a*s given only with respect to observed 
realism within five degrees of perfect realism. The proportion 
attaining defined realism is then the best index of the effect- 
iveness of explicit feedback, and the data do support the use 
of a realism graph fo:: students having relative difficulty with 
the problems. 

While the lack of overall gains may be related to inap- 
propriate difficulty levels or the absence of explicit feedback 
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in some cells of the design, a more likely explanation is the 
brevity of the training* Tho modification of long-established 
tendencies probably requires a period of months. It is 
possible that something of the effect of a longer period may 
be achieved in just three weelcs if the training is concentrated, 
but the evidence of this study indicates that nine 20-25 
minute sessions are not enough. In fact, it is surprising; 
that an average of 7-8 items per session provided enough 
training to be associated with any change in observed realism, 
e.g. the reduced standard deviations and the modest gain in 
the ntimber attaining defined realism. 

Time constraints also weakened the effect of the realism 
graph, because some students were* not able to stay Ipng enough 
at the end of a session to viev/ their graphs. The rush to 
board the bus tended to interfere with the process of receiv- 
ing explicit feedback both in terms of viewing the graph and 
listening to experimenter interpretations of the graph. With 
full and frequent exposure to realism graphs over several 
months, students may be able to develop assessment strategies 
that substantially alter established tendencies. 

Contrary to expectations, team problem-solving did not 
appear to support realism training. Especially when working 
with relatively difficult problems (second posttest), students 
found realism hard to achieve while interacting with their 
peers. One of the questions raised by this experiment is why 
team study is so closely associated with over confidence. 

Even v/ithout team study, however, seventh graders 
clearly tend to exaggerate the value of their information. 
This key finding agrees with Savage's observation, quoted 
earlier, that our culture, especially pur schooling, encouraged 
us to confuse belief v/ith certainty, lliough we have no assur- 
ance that realism can be taughc, it would seem that programs 
for realism training merit further study. 
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APPENDIX A. 
SCoRule Procedures Test* 



NAME TEACHER 



A BCD 

a. 76 76 60 60 

b. 70 70 70 70 

c. 95 66 0 0 • 

d. 100 0 0 0 

e. 95 43 43 43 
f- 85 85 0 0 

Which of the above SCoRule settings would be best if... 

1. you are sure that A is the correct choice, (Really surei) 

2. you are alir.ost sure that A is correct but B might 
be correct. (You know that C and D are wrong.) 

3. you are not sure which choice is correct but you 
have reason to think that A or B are more likely 
to be correct than C or D. 

4. you have no knowledge to help you make a choice. 

5. you arc almost sure that A is correct but if it 
isn't you have no idea which of the others is. 

6. you will just have to gut^ss. 

(Write a little Letter in each of the blanks.) 
*Scoring: 2 points per question^ 1 point for '*f'* on question 3. 
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ComputatLonal Procedures for 
Calculating Pealisiu during 
Trctining Sessions* / 



1. Student's responses = S{X) = S(A). S{B). S(C), S(D)** 

a* Each X is^ a one to three digit number ( 0<^Xj{100) 
which is input separately* 

b. Each X is a SCoRulc value (log score) . 

c. Order^ of input proceeds from first value (A) thru 

last value (D) • 

/ // 

2. Transformation from log score, S('X), to probability, p(X) 

a. ^6 < pj.(X) ^1.00 / >. 

b. /p(x) = anti logio ( S(X) - 100 ] 

/ V 50 / . • 



Validity check on responses , 
a. 



1,00 - I p {x)\L^ 

b. - .04*** 

4. Preparing student's response, p(X),.for curve fitting. 

1 - p(X) if X is correct answer 

a. d(X) =< -p(X)ifXis incorrect answer 

b. PD = ^ p{X) d(X) 

c. = ^ P(X)2 

**d. a^ - number of possible ansv/Ci" to ith item ~ 4 

5. Accumulating response data over items 

a. K = number of items answered in the session 
k 
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b* > PD and ^ p are sums accumulated item 

T 1 , i-l 

by item, i.e.S PD = <* PD + PD; is computed at the 

1 1. 
end of item i. 

k 

^*c. ;t a. = K * 4 
1 
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Curve fitting at end of session 



**a. 



P = 



= .25 



b. 



slope b = 




- Kp 



7. Category assignment for feedback. 

a. A slopeof ,82 is five degrees below b=1.00.^ 

b. A slope of 1.20 is five degrees above b-1.00. 



c* "Realistic" if .82 < b ^ 1.20. 

d. Overconfident" if b c -82. 

e. "Underconfident" if b > 1.20. 

♦Computational procedures for DEC-TEST, which was used to 
analyze the data, are presented in Brennan (1973). 

**Only four alternative items were used in this experiment. 

***This toleraji^ce level was arbitrary? it should not be 
confused wi|b^Ti the range of -tolerance for defined 
realism (7 above). 




f 
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APPENDIX 



Sujnomary Data for Tests 
And Training Sessions 



NOTE: Unless otherwise indicated, the standard 
deviation is shown below the mean, and 
the number of Ss and mean number of items 
are hyphenated on the line below the 
standard deviation. 



Example: 



8.32 (X) 
6.51 (sd) 
13 - 7.2 (N-K) 
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C-1 Cell Means for Angles of Deviation of Observed Realism from 
Perfect Realism. 



Low Class 
Treatment Group 



High 'Class 
Treatment Group 



TOTAL 
SAMPLE 



Pretest* 



18.47 
12.93 
8-18 .8 



19.05 
9.05 
9-19.0 



15.76 
17.14 
6-22.3 



10.84 
6.26 
7-21.6 



9.67 
8.48 
9-18.9 



7.54 
7.10 
10-23. 



13.31 
11.34 
49-20.6 



Sessions 
1 



10.98 
13.45 
6-5.5 



30.11 
27.43 
6-5,7 



42.37 
16.18 
2-8.5 



4.00 20.34 7.55 , 17.77 

5.65' 21.27 7.47 .,21.48 

6-4.8 8-5.6 2-6 ..0 30-5.7 



15.24 
18.58 
6-8.0 

13,27 
10.98 
8-8.8 



21.23 
20.84 

6- 7.3 

27.30 
20.30 

7- 6.4 



15.76 8.46 20.72 15.77 

.00 6.85 17.15 -00 

1- 4.0 6-6.3 6-5.2 1-5*0 

3.80 7.84 • 16.63 19.74 

1.37 8.60 13.71 .01 

2- 9.5 7-5.0 8-5.8 2-6.0 



15.54 
17.59 
26-6.5 

15.65 
14.97 
34-6.^ 



10.14 
11.70 
6-8.7 



12.77 
10.51 
6-6.2 



9.13 
2.07 
2-6.0 



9.35 9.30 15.31 10.72 

7.87 5.41 11.12 9.20 

7-4.6 7-4.9 3-7.0 31-6.1 



21.10 
18.73 
8-10.1 



35.00 
32.18 
6-7.5 



13.60 
2.16 
2-8.5 



11.62 11.89 /.93 17.83 

11.38 9.66 5.77 19.94 

7-8.1 8-7.4 3-11.0 34-8.6 



21.70 
22.75 
7-8.7 



21.21 
18.34 
9-7.9 



10.36 11.40 6,75 14.44 15.77 

10.34 4.88 3.12 8.31 15.96 

2-5.5 7-9.6 5-8.6 3-9.0 33-8.5 



19.94 
21.09 
6-6.2 



24.68 
27.39 
9-6.4 



18.98 
10.46 
2-7.0 



12.76 
6.39 
4-9.0 



7.59 
10.57 
3-9.0 



12.25 
3.95 
3rl0.0 



18.16 
20.32 
27-7.5 



8 



32.50 
23.99 
4-6.3 



26.97 
26.51 
5-6.2 



25.16 
13.96 
2-6.5 



9.17 13.65 14.01 19.22 
3.42 3.47 1.38 18.18 

6-10.5 5-9.4 3-8.3 25-8,2 



Posttests 



7.52 
5.14 
6-7.7 



22.59 
22.36 
8-7.5 



.65 
.64 
2-5.0 



9.00 
7.41 
7-9.1 



7.71 
6.39 
5-9.6 



12.85 
9.55 
3-8.7 



11.85 
14.43 
31-8.2 



13.57 
8.11 
8-32.1 



11.41 
11.52 
9-27. 



17.83 
10.48 
6-34.3 



8.22 
4.51 
7-36.4 



10.13 
5.73 
9-29.3 



8.89 
6.66 
10-32.7 



11.34 
8.68 
49-31.5 



2 18.69 26.11 26.90 16.06 13.04 19.04 19.32 

8.09 ^ 5.05 13.17 2.81 3.92 3.86 8.33 

7-20.0' 5-15.8 6-19,8 7-17.4 8-18.9 9-20.3 42-18.9 

* About five minutes of the 45 minute class period was spent completing 
the SCoRules Procedures Test. 
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C-2. Marginal Means for Angles of Deviation of 
Observed Realism from Perfect Realism 



Pretest* 



Sessions 



Class 
Low High 



17.99 
12.98 
23-19.8 



23.67 
23.96 
14-^6.0 



9.17 
7.53 
26-21.2 



12.61 
17.50 
16-5.4 



Treatment Group 
1 2 



14.91 
11.04 
15-20.1 



7.49 
10.89 
12-5.2 



14.36 
9.94 
18-18.9 



24.52 
24.59 
14-5.6 



10.62 
12.55 
16-22.8 



24.96 
21.49 
4-7.3 



TOTAL 
SAMPLE 

13.31 
11.34 
49-20.6 



17.77 
21.48 
30-5.7 



18.04 
19.20 
13-7.4 

17.93 
17.22 
17-7.9 



13.03 
15.42 
13-5.7 

13.37 
11.89 
17-5.5 



11.85 
14.41 
12-7.2 

10.73 
10.30 
15-70 



19.19 
20.88 
12-6.3 

21.61 
17.91 
15-6.7 



15.77 
.01 
2-4.5 

11.77 
8.03 
4-7.8 




15.65 
14.97 
34-6.7 



11.12 
10.43 
14-7.2 



10.38 
8.04 
17-5.1 



9.71 
9.83 
13-6.5 



10.90 
8.35 
13-5.5 



12.84 
9.23 
5-6.6. 



10.72 
9.20 
31-6.1 



25.38 
25.01 
16-8.9 



11.12 
9.97 
18-8.3 



16.68 
16.43 
15-9V2 



21.80- 

25.00 

14-7.4 



10.20 
5.44 
5-10.0 



17.83 
19.94 
34-8.6 



20.20 
19.84 
18-7 . 9 



10.46 
6.03 
15-9.1 



16.55 
17.24 
14-9.1 



16.05 
16.36 
14-8.1 



12.81 
9.39 
5-7.6 



15.77 
15.90 
33-8.5 



22.34 
23.94 
11-6.4 



11.06 
7.73 
10-9.3 



17.07 
17. 2p 
10-7.3 



20.41 
25.40 
12-7.1 



14.94 
8.00 
5-8.8 



18.16 
20.32 
?7-7.5 



28.65 
23.94 
11-6.3 



11.81 
3.87 
14-9.6 



18.50 
19.18 
10-8.8 



20.31 
20.04 
10r7 . 8 



•18.47 
10.43 
5-7.6 



19.22 
18.18 
25-8.2 



Posttests 
1 



14.20 
18.30 
10-7.3 



13.84 
10.48 
23-30.9 



9.34 
7.80 
15-9.2 



9.14 
5.88 
26-32.5 



8.32 
6.51 
13-8.5 



11.08 
7.19 
15-34.1 



16«80 
19.38 
13-8.3 



10.77 
9.13 
18-28.4 



7.97 
9.52 
5-7.2 



12.24 
9.36 
16-33.3 



11.85 
14.43 
31-8.2 



11.34 
8.68 
49-31.8 



23.52 
10.26 
18-18.8 



16.17 
4.40 
24-19.0 



17.37 
6.20 
14-18.7 



18.07 
7.73 
13-17.7 



22.22 
9.67 
15-20.1 



19.32 
8.33 
42-18.9 



♦About five minutes of the 45-minute class period was spent completing 
the SCoRule Procedures Test. 
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C-3 Cell Means and Standard Deviations for 
Estimated Proportion Correct* 



Low Class 
Treatment Group 



High Class 
Treatment Group 



* See Appendix C-1 for and K in each cell< 
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TOTAL 





1 


2 


3 


1 


2 


3 




Pretest 


.53 


.46 


.53 


.73 


.64 


.74 


.61 




.18 


.19 


.22 


.07 


.18 


.15 

♦ 


.20 


Sessions 
















1 


.54 


.39 


.26 


.73 


.49 


.46 


.51 




.16 


.22 


.09 


.14 


.28 


.11 


.24 


2 


.57 


.61 


. .56 


.73 


.54 


.50 


.61 




.22 


.31 


.00 


1 


.25 


.00 


.24 


3 


.44 


.41 


.79 


.66 


.49 


.55 


.52 




.16 


.18 


.09 


.14 


.22 


.05 


.20 


4 


.55 


.53 


.70 


.69 


.58 


.61 


.60 




.17 


.08 


.05 


.16 


.26 


.21 


.19 


5 


.41 


.37 


.55 


.68 


.50 


.67 


.51 




.17 


.19 


.02 


.19 


,14 


.08 


.20 




.46 


.52 


.68 


.69 


.70 


.65 


.59 




.19 


.20 


■.18' ■ 


.10 


.12 


.13'. 


.19 


7 


.47 


.45 


^ .41 


.63 


.60 


.64 


.51 




.15 • 


.27 


.16 


.11 


.23 


.05 


.22 


8 


.33 


.43 


.45 


.71 


.57 


.64 


.54 




.28 • 


.31 


.13 


.11 


.14 


.04 


.23 


9 . 


.54 


.44 


.85 


.72 


.50 


.70 


.60 




.21 


.21 


• .15 


.18 


.17 . 


.14 


.23 


Posttests 
















1 


.53 


.56 


.52 


.70 


.64 


.72 


.62 




.12 


.18 


.14 


.07 


.16 


.13 


.16 


2 


.42 


.41 


.35 


.55 


.59 


.54 


.49 




.09 


.05 


.06 


.08 


.08 


.07 


.11 



4<. 



\ 



C-4 M<.r^jinal Means and Standard Deviations 
for fiktiinated Proportion Correct* 



Pretest 

Sessions 
1 



Lev; 

• 50 
.20 



.44 
.21 

.59 
.26 

.47 
.20 



Class 




Treatment Group 



.55 .66 
.21 .20 



.45 .36 

.26 .14 

.57 .53 

.28 .03 

.45 .67 

.21 .14 



TOTAL 

SAMPLE 

.61 
.20 



.51 
.24 

.61 
.24 

.52 
.20 



.56 
.14 



.63 

»22 



.62 
.18 



.56 
.20 



.65 
.17 



.60 
.19 



.41 
.18 



.60 

;i8 



.54 
.23 



.45 
.18 



.62 
.09 



.51 
.20 



.5i 
.20 



.68 
.11 



.57 
.19 



.58 
.19 



.66 

.15 



.59 
.19 



.45 
.23 



.63 
.14 



.53 
.16 



.49 
.27 



.55 
.16 



.51 
.22 



.40 
.26 



.65 
.13 



.56 

.25 



.50 
.25 



.57 
.13 



.54 
.23 



.53 
.24 



.67 
.18 



.76 
.16 



.64 
.21 



.60 
.73 



.60 
.23 



Posttests 
1 



.54 
.25 



.69 
.13 



.61 
.13 



,60 
.17 



.65 
.16 



.62 
.16 



.39 
.08 



.56 
.08 



.48 
.11 



.52 
.11 



.46 
.12 



.49 

ill 



* See Appendix C-2 for N and K information. 



42. 



Cell Means and Standard Da'/iations for Mean Log 

Scores (Average SCoRule Val^ue for Correct Alternatives)* 



Low Class High Class TOTAL 

Treatment: Group Treatment Group SAMPLE 

1 2 3 12 3 



It JL t= O \m 


/ V/ • V/O 


1 1,* £,0 


7*1 Q? 


78 02 


75, 68 


80.76 


75.10 




9,58 


7.33 


8,74 


3.64 


10.44 


11.63 


9.93 


Sessions 
















1 


75,49 


58.13 


63.17 


83.54 


64.10 


76.73 


69.85 




6.10 


17.48 


1.17 


9.95 


22*58 


6.12 


17 .86 






• 5 


• 0 


.7 


• 5 


. 5 


.4 


- 2 


73.98 


77.58 


67.50 


80.62 


70.05 


68'. 00 


74.96 




13.11 


13.13 


.00 


11.70 


11.88 


• 00 


12« 74 




1.2 


1.0 


.0 


. 3 


1« 5 


.0 


. 9 


3 


70.72 


71.68 


85.12 


79.74 


68.54 


61.43 


72.56 




8. 59 


5. 63 


2.39 


11. 11 


17.91 


1.43 


1^ . 54 




1.1 


.7 


1.0 


.0 


. 6 


. 0 


* o 


4 


60.14 


71.85 


77.71 


77.08 


77.13 


70.79 


76.10 




6.82 


6.54 


3.71 


12.50 


9.27 


17.83 


10. 6C 




.3 


.8 


.0 


.0 


.7. 


.0 


.4 


5 


65.96 


65.90 


71.10 


77.23 


69.25 


80.10 


70.59 




12.81 


12.83 


3.32 


10.83 


9.51' 


8.97 


12.13 




1.2 


1.0 


2.5 


• 4 


«4 


• 0 


.0 


6 


68.27 • 


71.49 


76.10 


74.74 


79.73 


70.71 


72.95 




7.21 


8.14 


17 . 90 


8.02 


5.65 


13.52 


9.80 




.7 


.4 


3.0 


■ .1 


.2 


.0 


.5 


* 7 


t 

68.02 


69.74 


74.77 


70.06 


77.09 


70.5a 


70.69 




14.61 


19.86 


2.23 


11.29 


15.72 


5.47 


15.38 




.0 


.2 


2.0 


• 0 


.0 


.0 


.2 


o 
o 


D J . O / 




no 


7n 1 o 




70 11 


UO . CJ v> 




17.16 


21.11 


13.44 


5.96 


5.40 


2.67 


15-22 




.0 


.8 


2.5 


1.0 


.0 


• 0 


• 6 


9 


78.93 


63.02 


87.19 


79.20 


76.53 


72.42 


74.40 




9.G7 


24 . oU 


12 .81 


X2 . yJb 


7. 91 




i / . 01 




.3 


.3 


2.5 


• 6 


• 0 


.0 


• 4 


Posttcsts 
















1 


69.87 


75.17 


66.19 


78.61 


76.64 


79.10 


74.77 




7.47 


\ 9.76 


11.91 


6.83 


7.26 


11-37 


10.31 


2 


66.45 


^58.49 


57.57 


67.16 


70.59 


62.38 


64.27 




4.61 


9.09 


12.39 


3.50 


4.74 


5^52 


8.31 


♦See Appendix C-l 


for K and 


K in each 


cell. 









♦♦Mean number of computer validity chccks# i.e. mean number of times in which 
an item's probability values (anti log of SCoRule values) did not sum to 1.00 
within a .04 tolerance.* S then re-entered values. 



43. 



C-6. Marginal M'jans and Standard Deviations for 
Mean Log Scores* 



Pretest 



Class Treatment Group TOTAL 

SAMPLE 

Low Hi^h 

71.52 78.26 73.78 73.45 78.19 75.10 

8.67 9.90 8.42 9.29 11.14 • 9.93 



Sessions 

1 66.29 72.97 79. 51 61.54 69i95 69.85 
15.02 19.49 9.93 20.76 8.09 17.86 

.2** .6 .3 .5 * .3 . .4 

2 75.14 74.77 77.30 73.81 67.75 " 74.96 
12.91 12.57 12.86 13.07 .25 12.74' 

1.0 .9 .8 1.3 .0 .9 



3 


72.81 
8.30 
.9 


72.32 
15.67 
.3 


74.93 
10.83 
.6 


70.01 
13.73 
.7 


73.28 
12.01 
.5 


72.56 
12.54 
.6 


4 


^6.24 
7.44 
.5 


75.99 
12.71 
.3 


78.49 
10.39 
.2 


74.69 
8.54 
.8 


73.56 
14.41 
.0 


76.10 
10.66 
• 4 


5 


66.58 
12- 17 
1- 3 


74.16 
10. 93 
. 3 


71.22 
8.28 
.4 


67.82 
11- 18 
.6 


76.50 
8. 50 
1.0 


70.59 
12. 13 
.8 


6 


70.75 
9.73 
.8 


75.60 
9.38 
.1 


71.51 
8.28 
.4 


74.43 
8.34 
^ .4 


72.87 
15.65 
1.2 


72.95 
9.88 
.5 


7 


69.72 
16.99 
.4 


72.33 
12.00 
.0 


60.84 
, 13.42 
.0 


71.57 
19.17 
• 2 


72.26 
4.91 


70,69 
15; 30 
.2 


8. 


62.10 
19.92 
.8 


74.10 
6.29 
.4 


68.32 
16.00 
.6 


70.74 
15.43 
.4 


•66.16 
10.17 
1.0 


68«86 
15.22 
.6 


9 


72.01 
21.27 
.6 


76.96 
12.08 
.3 


79.07 
11.02 
.5 


60.23 
21.18 
.2 


78.33 
16.44 
1.0 


74.40 
17.61 
• 4 



Posttesta 

1 ' 70.99 78.12 73.95 75.91 74.26 74.77 

10.35 9.04 8.40 8.64 13.16 10.31 

2 61.28 66.51 66.80 65.93 60.46 64.27 

9.98 5.88 4.11 8.96 9.23 8.31 

♦See Appendix C-2 for N and K inforination 
**Mean number of computer validity checks (see Appendix C-5) . 



44. 



APPENDIX 

> 

Means and Standard Deviations for Angles of Deviation 
of Observed Realism from Perfect Realism 
On Easy and -Hard Items 
During Training Sessions 



• 



Session 



N 
26 



Easy 
K* 

2 . 56 
1.22 



X* 

15.58 
18.62 



N 
25 



Hard 
K 

3.12 
1.17 



X 

27.26 
26.87 



26 



3.35 
1.41 



10.29 
14.49 



26 



3.46 
1.17 



"^4.53 
23V80 



28 



3.50 
1.37 



21.55 
25.36 



32 



3.53 
1.11 



16.30 \ 
19.97 



25 



3.32 
1.22 



7.69 
11.95 



29 



3.59 
1.18 



21.19 
21.48 



35 



4.26 
1.50 



13.92 
19.87 



33 



4.36 
1.45 



30.67 
26.69 



36 



4.06 
1.33 



7.42 
9.70 



32 



4.34 
1.43 



27.67 
22.21 



7 



25 



3.72 
1.40 



11.62 
19-48 



25 



3.92 
1.44 



?:.25 
24.70 



25 



4.24 
1.51 



14.81 
13.85 



24 



3.63 
1.41 



29-08 
23.71 



34 



3.97 
1.61 



8.38 
15.65 



32 



4.13 
1.43 



32.13 
27.69 



*3tnndard deviations are shown below means. 



45. 



ERIC 



